Storage managing method and storage management device

ABSTRACT

The disclosed video processing device contains: a video acquisition unit that acquires surroundings information including video taken of the surroundings of a vehicle; a line-of-sight acquisition unit that acquires the origin and direction of the line of sight of the driver of the aforementioned vehicle; a line-of-sight video generation unit which generates, from the surroundings information, line-of-sight video corresponding to the origin of the line of sight; a blocking-information computation unit that computes, on the basis of the origin of the line of sight, blocking information including video or a region of the body of the aforementioned vehicle that blocks the driver&#39;s line of sight; and a display-video generation unit that generates display video on the basis of the line-of-sight video and the blocking information.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of International application No.PCT/JP2010/058774, filed on May 25, 2010, the entire contents of whichare incorporated herein by reference.

FIELD

The present invention is related to a picture processing device, apicture processing method, and a picture processing program forgenerating pictures observed by a driver.

BACKGROUND

In the society dependent on cars, it is an important problem to promotesafety and reduce the number of accidents, and various measures aretaken to solve the problem. For example, one of the measures is to allowa driver to learn to know using pictures the case in which an accidenteasily takes place. Concretely, for example, according to the patentdocument 1, a picture of an accident is acquired using a drive recorderloaded into the vehicle, and the picture of an accident is replayed toeffectuate traffic safety education. For example, when the driverecorder detects an impact of a car crash or dangerous driving such as asudden braking, a sharp turn of the steering wheel, and so on, a viewahead of the driver's vehicle and the driving state of the driver'svehicle are recorded.

However, the picture acquired by the above-mentioned drive recorder isonly the view ahead of the vehicle, and the picture which may beconfirmed by viewer is also limited to the view ahead of the vehicle.Therefore, for example, when the driver looks at the right or left side,the picture viewed by the viewer is different from the picture actuallyviewed by the driver.

It is effective for traffic safety education to analyze the cause of thedangerous driving such as what situation the driver has actuallyobserved, what situation observed by the driver has incurred thedangerous driving, and so on.

Patent Document 1: Japanese Laid-open Patent Publication No. 2007-011148

SUMMARY

According to an aspect of the picture processing device of the presentinvention, the device includes: a picture acquisition unit whichacquires the peripheral information including the picture obtained byshooting the periphery of a driver's vehicle; a line-of-sightacquisition unit which acquires the line-of-sight origin and thedirection of the line of sight of a driver of the driver's vehicle; aline-of-sight picture generation unit which generates from theperipheral information a line-of-sight picture corresponding to theline-of-sight origin; a cutoff information calculation unit whichcalculates the cutoff information including the car body area or the carbody picture of the driver's vehicle which cuts off the line of sight ofthe driver based on the line-of-sight origin; and a display picturegeneration unit which generates a display picture according to theline-of-sight picture and the cutoff information.

The object and advantages of the invention will be realized and attainedby means of the elements and combinations particularly pointed out inthe claims.

It is to be understood that both the foregoing general description andthe following detailed description are exemplary and explanatory and arenot restrictive of the invention.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is an example of the connection among the driving pictureprocessing device, the information acquisition device, and the drivetraining terminal according to the first embodiment of the presentinvention;

FIG. 2 is an example of a hardware configuration of a driving pictureprocessing device, an information acquisition device, and a drivetraining terminal;

FIG. 3 is an explanatory view of the attaching position and the shootingrange of a peripheral information acquisition equipment;

FIG. 4 is an explanatory view of the attaching position of theline-of-sight detection equipment;

FIG. 5 is an explanatory view of the attaching position of theline-of-sight detection equipment;

FIG. 6 is an example of the appearance of a vehicle;

FIG. 7 is an explanatory view of an example of the area which may beconfirmed by a mirror;

FIG. 8 is an example of a block diagram of the functional configurationof each device according to the first embodiment;

Part (a) of FIG. 9 is an explanatory view (1) of an example of a methodof calculating the line-of-sight origin P and the line-of-sight vector,part (b) of FIG. 9 is an explanatory view of the pitch angle θβ, andpart (c) is an explanatory view of the azimuth θα;

FIG. 10 is an example of peripheral information DB;

FIG. 11 is an example of line-of-sight data DB;

FIG. 12 is an explanatory view (1) of the relationship between theeffective vision range and the mirror, and the visual mirrorconfirmation range which may be visually confirmed through a mirror, anexplanatory view (2) of the relationship between the effective visionrange and the mirror, and the visual mirror confirmation range which maybe visually confirmed through the mirror, and an explanatory view (3) ofthe relationship between the effective vision range and the mirror, andthe visual mirror confirmation range which may be visually confirmedthrough the mirror;

FIG. 13 is an example of the association between the line-of-sightorigin p and the line-of-sight vector, and the car window vision area onthe 3-dimensional projection surface;

FIG. 14 is an example of the association between the line-of-sightorigin P and the line-of-sight vector, and the car window cutoffinformation;

FIG. 15 is an explanatory view of the relationship between the effectivevision range and the mirror, and the visual mirror confirmation rangewhich may be visually confirmed through the mirror;

FIGS. 16A and 16B are an example of the association for each car modelamong the mirror information, the line-of-sight origin P, the virtualline-of-sight origin VP, the mirror vision field angle θm, the mirrorvision area, and the mirror cutoff information;

FIG. 17 is an explanatory view (1) of the relationship between the carwindow picture and the mirror picture, and the display area of thedisplay on the 3-dimensional projection surface;

FIG. 18 is an explanatory view (2) of the relationship between the carwindow picture and the mirror picture, and the display area of thedisplay on the 3-dimensional projection surface;

FIG. 19 is an explanatory view (3) of the relationship between the carwindow picture and the mirror picture, and the display area of thedisplay on the 3-dimensional projection surface;

FIG. 20 is an example of the association between the line-of-sightorigin P and the line-of-sight vector, and each mirror display area;

FIG. 21 is an example of a car window picture DB;

FIG. 22 is an example of a mirror picture DB;

FIG. 23 is an example of cutoff information DB;

FIG. 24 is an example of a picture used as a car window display picture;

Part (a) of FIG. 25 is a car window picture including another vehicleand a walker when the driver looks ahead, and part (b) is a car windowdisplay picture obtained by combining the car window cutoff informationwith the car window picture in (a);

Part (a) of FIG. 26 is a car window picture including another vehicleand a walker when the driver looks diagonally right ahead, and part (b)is a car window display picture obtained by combining the car windowcutoff information with the car window picture in (a);

Part (a) of FIG. 27 is a car window picture including another vehicleand a walker when the driver looks diagonally left ahead, and part (b)is a car window display picture obtained by combining the car windowcutoff information with the car window picture in (a);

FIG. 28 is an example of a display picture obtained by superposing aback mirror picture on a back mirror display area 266B in part (b) ofFIG. 26;

FIG. 29 is an example of a display picture obtained by superposing aright mirror picture on a right mirror display area 266R in part (b) ofFIG. 26;

FIG. 30 is a flowchart of an example of the flow of the processperformed by the driving picture processing device according to thefirst embodiment;

FIG. 31 is an explanatory view of the positions of the fixed vision areaon the 3-dimensional projection surface, and the car window picture andthe mirror picture;

FIG. 32 is an explanatory view of the relationship between the carwindow picture and the mirror picture on the 3-dimensional projectionsurface, and the display area of the display;

FIG. 33 is an example of a display picture;

FIG. 34 is an example of a block diagram of the functional configurationof each device according to the variation example 2;

FIG. 35 is an explanatory view of an example of processing a displaypicture;

FIG. 36 is an example of a block diagram of a hardware configuration ofthe driving picture processing device according to the secondembodiment; and

FIG. 37 is an example of a block diagram of a functional configurationof the driving picture processing device according to the secondembodiment.

DESCRIPTION OF EMBODIMENTS First Embodiment

A driving picture processing device 100 according to the firstembodiment generates a picture actually observed by a driver of adriver's vehicle 300 during driving. For example, the driving pictureprocessing device 100 generates from the peripheral information aboutthe driver's vehicle 300 the line-of-sight picture corresponding to theline-of-sight origin p of the driver of the driver's vehicle 300 and thedirection of the line of sight. The peripheral information includes atleast the peripheral picture of the driver's vehicle 300, and includes,for example, the picture of an object such as a vehicle in addition tothe periphery of the driver's vehicle 300, the road, and so on.Furthermore, the driving picture processing device 100 calculates thecutoff information including the car body area and/or the car bodypicture of the driver's vehicle 300 which cuts off the line of sight ofthe driver based on the line-of-sight origin P and the direction of theline of sight of the driver. Next, the driving picture processing device100 generates the display picture having the line of sight of the driverat the center. Therefore, the 100 reflects on the display picture thearea which the driver has not observed due to the car body, andgenerates the picture around the driver's vehicle 300 which the driverhas actually observed by setting the line of sight of the driver at thecenter.

The car window line-of-sight direction from the line-of-sight origin Pof the driver through the window of the driver's vehicle 300 is includedin the direction of the line of sight, and the car window picturecorresponding to the line-of-sight origin P and the car windowline-of-sight direction are included in the line-of-sight picture. Inaddition, the driver may confirm the object at the back or the back sideof the driver's vehicle 300 through the mirror provided for the driver'svehicle 300. Accordingly, the line-of-sight picture may include themirror picture corresponding to the visual mirror confirmation rangevisible for the driver through at least one mirror of the driver'svehicle 300.

The cutoff information includes the car window cutoff information aboutthe body of the driver's vehicle 300 which cuts off the line of sight ofthe driver in the car window line-of-sight direction and/or the mirrorcutoff information about the body of the driver's vehicle 300 which cutsoff the mirror line-of-sight of the driver toward the visual mirrorconfirmation range.

In addition, the display picture includes the car window display pictureobserved by the driver through the window of the driver's vehicle 300and/or the mirror display picture observed through at least one mirror.The car window display picture is generated by combining the car windowpicture with the car window cutoff information, and the mirror displaypicture is generated by combining the mirror picture with the mirrorcutoff information.

Described below are the relationship among the driving pictureprocessing device 100 according to the first embodiment, an informationacquisition device 200 which acquires various types of information, anda drive training terminal 250, and the hardware configuration of eachcomponent.

(1) Relationship Among Driving Picture Processing Device, InformationAcquisition Device, and Drive Training Terminal

FIG. 1 is an example of the connection among the driving pictureprocessing device, the information acquisition device, and the drivetraining terminal according to the first embodiment of the presentinvention. FIG. 2 is an example of a hardware configuration of a drivingpicture processing device, an information acquisition device, and adrive training terminal.

The driving picture processing device 100 combines the line-of-sightpicture corresponding to the line-of-sight origin P and the direction ofthe line of sight with the cutoff information about the car body of thedriver's vehicle 300 which cuts off the line of sight of the driver, andgenerates a display picture which having the line of sight of the driverat the center. The information acquisition device 200 acquires theperipheral information about the driver's vehicle 300 and various typesof information such as the line-of-sight data of the driver of thedriver's vehicle 300. The drive training terminal 250 is used by aviewer such as the driver who receives safe drive training to view thedisplay picture generated by the driving picture processing device 100.

The driving picture processing device 100, the information acquisitiondevice 200, and the drive training terminal 250 are connected so thatvarious types of information may be transmitted and received. Theconnecting method may be, for example, an interface such as an SCSI(small computer system interface), a USB (universal serial bus), and soon, and a network such as the Internet, and so on.

(2) Hardware Configuration

(2-1) Driving Picture Processing Device

The driving picture processing device 100 includes, for example, a CPU(central processing unit) 101, ROM (read only memory) 102, RAM (randomaccess memory) 103, input/output equipment I/F 104, a communication I/F(interface) 108, an HDD (hard disk device) 110 a, R/W (read/write)equipment 110 b. These components are interconnected through a bus 109.

The input/output equipment I/F 104 is connected to an input/outputequipment such as a display 105, a mouse 106, a keyboard 107, and so on.

The ROM 102 stores various control programs relating to various types ofcontrol described later and performed by the driving picture processingdevice 100.

The RAM 103 temporarily stores various types of information such asperipheral information and line-of-sight data, and so on acquired fromthe information acquisition device 200. The RAM 103 also temporarilystores the information such as various flags depending on the executionof each type of control program.

The HDD 110 a is an auxiliary storage management device, and storesvarious types of information such as peripheral information,line-of-sight data, and so on acquired from the information acquisitiondevice 200.

The R/W equipment 110 b writes the various types of information to anexternal storage device, or reads various types of information,programs, and so on stored in an external storage device. The externalstorage may be an external HDD, a computer-readable recording medium,and so on.

The CPU 101 develops various control programs stored in the ROM 102 onthe RAM 103, and perform various types of control described later.

The communication I/F 108 communicates a command or data between, forexample, the information acquisition device 200 and the drive trainingterminal 250 based on the control of the CPU 101.

The bus 109 is configured by, for example, a PCI (peripheral componentinterconnect) bus, an ISA (industrial standard architecture) bus, and soon, and these components are interconnected.

(2-2) Information Acquisition Device

The information acquisition device 200 includes, for example, a CPU 201,ROM 202, RAM 203, an input/output equipment I/F 204, a communication I/F207, an HDD 209 a, and R/W equipment 209 b. These components areinterconnected.

(a) Input/Output Equipment I/F

The input/output equipment I/F 204 is connected to a peripheralinformation acquisition equipment 205, a line-of-sight detectionequipment 206, and so on. The information detected by the peripheralinformation acquisition equipment 205 and the line-of-sight detectionequipment 206 is output to the RAM 203, the CPU 201, the communicationI/F 207, and so on.

(b) Peripheral Information Acquisition Equipment

The peripheral information acquisition equipment 205 acquires theperipheral information about the periphery of the driver's vehicle 300.In the present embodiment, it is assumed that the peripheral informationacquisition equipment 205 acquires the peripheral picture around thedriver's vehicle 300. The peripheral picture includes an object, forexample, the people, a vehicle, and so on around the driver's vehicle300, and a road and so on. The peripheral information acquisitionequipment 205 includes an image pickup device such as a CCD (chargecoupled device) camera, a CMOS (complementary metal oxide semiconductor)camera, and so on.

FIG. 3 is an explanatory view of the attaching position and the shootingrange of a peripheral information acquisition equipment. The peripheralinformation acquisition equipment 205 is configured by, for example,four cameras, that is, a forward camera 205 a, a right camera 205 b, aleft camera 205 c, and a backward camera 205 d as illustrated in FIG. 3.The forward camera 205 a is attached to the center of the forward bumperof the driver's vehicle 300, and shoots a forward object of the driver'svehicle 300. The backward camera 205 d is attached at the center of thebackward bumper of the driver's vehicle 300, and shoots a backwardobject of the driver's vehicle 300. The right camera 205 b is attachedat the center of the right of the driver's vehicle 300, and shoots theright side of the driver's vehicle 300. The left camera 205 c isattached at the center of the left side of the driver's vehicle 300, andshoots the left side of the driver's vehicle 300.

Each of the cameras 205 a through 205 d is a camera using asuper-wide-angle lens having the angle of view of 180°. Thus, asillustrated in FIG. 3, the forward camera 205 a shoots the forward area210 of the driver's vehicle 300, the right camera 205 b shoots a rightarea 211 of the driver's vehicle 300, the left camera 205 c shoots aleft area 212, and the backward camera 205 d shoots a backward area 213of the driver's vehicle 300. The shooting area of each of the cameras205 a through 205 d is configured so that the area overlaps the areashot by the adjacent cameras.

The attaching position and the attachment angle of each of the cameras205 a through 205 d and the characteristics of the camera such as thedistortion correction value, the focal distance, and so on of the lensof the camera are corrected, that is, calibrated so that they may beapplied to the spatial coordinate system having the center point O ofthe driver's vehicle 300 as the origin. By performing the calibration,the picture shot by each of the cameras 205 a through 205 d may beincorporated into the spatial coordinate system. The spatial coordinatesystem is expressed by the X, Y, and Z coordinates. For example, thecenter point O is defined by the center of the driver's vehicle 300which refers to the half values of the width and the length of thedriver's vehicle 300, and is expressed by (X, Y, Z)=(0, 0, 0). Yindicates the forward direction, X indicates the direction orthogonal tothe forward direction, and Z indicates the direction of the height.

It is preferable that each of the cameras 205 a through 205 d isattached to the center of each of the front side, the right side, theleft side, and the back side of the driver's vehicle 300. However, it isaccepted that the shooting area of each of the cameras 205 a through 205d partially overlaps the shooting area of the adjacent camera, and theattaching position of each of the cameras 205 a through 205 d is notspecifically limited. For example, the right camera 205 b and the leftcamera 205 c may be attached to the left and right door mirrors. Inaddition, it is accepted that the shooting area of each camera partiallyoverlaps another's and the area around the driver's vehicle 300 may beshot at 360°, and the number of cameras is not limited to four.

Each of the cameras 205 a through 205 d shoots 30 frames per second. Thepicture data shot by the peripheral information acquisition equipment205 configured by the cameras 205 a through 205 d is stored on the RAM203 through the input/output equipment I/F 204.

By shooting a picture by each of the cameras 205 a through 205 d asdescribed above, a picture observed by the driver may be generated.

The peripheral information acquisition equipment 205 does not acquirethe peripheral information constantly during the operation, but mayrecord the peripheral information only on a specific occasion such aswhen the tracing drive is detected upon detection of, for example,dangerous driving.

(c) Line-of-Sight Detection Equipment

The line-of-sight detection equipment 206 detects the line-of-sightinformation such as the face, eyeball, iris, and so on of a driver.

FIGS. 4 and 5 are explanatory views of the attaching position of theline-of-sight detection equipment. The line-of-sight detection equipment206 is configured by an image pickup device such as a CCD camera, a CMOScamera, an infrared camera, and so on which are capable of acquiring theline-of-sight information about the driver.

The line-of-sight detection equipment 206 is provided on, for example, adashboard 301 of the driver's vehicle 300 as illustrated in FIGS. 4 and5. In this case, the line-of-sight detection equipment 206 is attachedat a specified angle on the dashboard 301 near a handle 302 so that theface, the eyes, and so on of the driver may be detected from the frontside and so that the face, the eyes, and so on may be shot withoutcutoff by the handle 302. However, if the face, the eyes, and so on ofthe driver are detected, the attaching position, the attachment angle,and so on are not restricted.

The characteristics of the line-of-sight detection equipment such as theattaching position, the attachment angle, and so on of the line-of-sightdetection equipment 206 are corrected, that is, calibrated so that thecharacteristics may be applied to the spatial coordinate system in whichthe center point O of the driver's vehicle 300 is an origin.

The line-of-sight detection equipment 206 shoots 30 picture frames persecond, and the shot picture data is stored on the RAM 203 through theinput/output equipment I/F 204.

A line-of-sight 150 may be detected based on the picture of the face,the eyeball, the iris, and so on of the driver detected by theline-of-sight detection equipment 206. If the line-of-sight 150 of thedriver is detected, the direction in which the driver has visuallyconfirmed is known.

As illustrated in FIGS. 4, 5, and 6, a vehicle is configured by a carbody structure such as the dashboard 301, the handle 302, windows,mirrors, pillars, and so on. FIG. 6 is an e of the appearance of avehicle. The driver's vehicle 300 has windows such as a front window306F, a right window 306R, a left window 306L, a back window (notillustrated in the attached drawings), and so on, and each window issupported by pillars. The pillars are, for example, a front pillar 307Flocated above the front window 306F, a right pillar 307R and a leftpillar 307L located at the right and left of the front window 306F, aback pillar 307B at the back of the car body, and so on.

In addition, a vehicle 303 provided for the driver's vehicle 300 may bedoor mirrors 303L and 303R attached near the left and right doors of thedriver's vehicle 300, a back mirror 303B provided in the driver'svehicle 300, a fender mirror provided on the hood of the driver'svehicle 300 as illustrated in FIGS. 5 and 6, and so on.

It is estimated that, for example, the driver has visually confirmed theperiphery of the driver's vehicle 300 through the front window 306F ifthe detected direction of the line-of-sight 150 is the forwarddirection. In addition, if the direction of the line-of-sight 150 isheaded for the mirror 303, it is estimated that the driver has visuallyconfirmed in the backward and diagonally backward directions through themirror 303. FIG. 7 is an explanatory view of an example of the areawhich may be confirmed by the mirrors. The driver of the driver'svehicle 300 may view a left mirror area 304L through the left doormirror 303L. The driver may also view a right mirror area 304R throughthe right door mirror 303R. The driver may also view a back mirror 304Bthrough the back mirror 303B.

(d) ROM, RAM, HDD, R/W Equipment, Communication I/F

The ROM 202 stores various control programs executed by the informationacquisition device 200.

The RAM 203 temporarily stores various control programs in the ROM 202,various flags, and various types of information received from theperipheral information acquisition equipment 205 and line-of-sightdetection equipment 206.

The communication I/F 207 transmits and receives data such as peripheralinformation, line-of-sight data, various commands, and so on to and fromthe driving picture processing device 100 under the control of the CPU201.

The HDD 209 a is an auxiliary storage device, and stores various typesof information acquired by the information acquisition device 200.

The R/W equipment 209 b writes the various types of information to anexternal storage device, or reads various types of information andprograms stored on the external storage device. The external storagedevice may be, for example, an external HDD and a computer readablerecording medium.

(e) CPU

The CPU 201 develops various types of control programs stored on the ROM202 to the RAM 203, and performs various types of control. For example,the CPU 201 controls the peripheral information acquisition equipment205, the line-of-sight detection equipment 206, and so on by executingvarious control programs, and starts acquiring various types ofinformation such as peripheral pictures.

(2-3) Drive Training Terminal

The drive training terminal 250 is used by a user who receives safedrive training, and the display picture generated by the driving pictureprocessing device 100 may be viewed on the terminal.

The drive training terminal 250 includes, for example, a CPU 251, ROM252, RAM 253, input/output equipment I/F 254, a communication I/F 258,an HDD 260 a, and R/W equipment 260 b. These components areinterconnected through a bus 259.

(a) Input/Output Equipment

The input/output equipment I/F 254 is connected to input/outputequipment such as a display 255, a mouse 256, a keyboard 257, and so on.The input/output equipment I/F 254 accepts an instruction to display thedisplay picture from a user. A speaker for outputting voice and so onmay be connected to the input/output equipment I/F 254.

(b) Display

The display 255 may be of any type so far as a display picture may beoutput. For example, the display 255 may be a flat display device, abended or flexible display device, and a combination of a plurality ofdisplay devices.

The display area of the display 255 includes a car window display area265 in which a car window display picture observed by the driver througha window 306 is displayed. Furthermore, the display area of the display255 may include a mirror display area 266 in which the mirror displaypicture observed by the driver through the mirror 303 is displayed. Theposition of each mirror display area 266 corresponding to the displayarea of the display 255 depends on the line-of-sight origin P and thedirection of the line of sight. Therefore, various correspondence tablesDB 131 described later stores the occupation position of each mirrordisplay area in the display area for each line-of-sight origin P anddirection of the line of sight, and the car window display picture andthe mirror display picture are displayed on the display 255 based on thecorrespondence.

(c) Others

The CPU 251 develops on the RAM 253 the various control programs storedon the ROM 252, acquires the data of the display picture from thedriving picture processing device 100, and outputs the data to thedisplay 255 and so on. The HDD 260 a stores, for example, various typesof information acquired from the driving picture processing device 100.Other configurations are substantially the same as those of the drivingpicture processing device 100, and the explanation is omitted here.

(3) Functional Configuration

Described next is the functional configurations of the driving pictureprocessing device 100, the information acquisition device 200, and thedrive training terminal 250.

FIG. 8 is an example of a block diagram of the functional configurationof each device according to the first embodiment. The connection line ofeach functional unit illustrated in FIG. 8 is an example of the flow ofdata, and does not describe the entire data flow.

Described first below is the functional configuration of the informationacquisition device 200.

(3-1) Information Acquisition Device

The information acquisition device 200 functions as each function unitdescribed later by executing a program with each hardware configurationcooperating with one another in the information acquisition device 200.

The functional unit of the information acquisition device 200 includes,for example, a peripheral information acquisition unit 220, aline-of-sight detection unit 221, a transmission/reception unit 222, anacquired data DB 223, and so on.

(3-1-1) Peripheral Information Acquisition Unit

The peripheral information acquisition unit 220 acquires the peripheralpicture shot by the peripheral information acquisition equipment 205configured by the forward camera 205 a, the right camera 205 b, the leftcamera 205 c, and the backward camera 205 d illustrated in FIG. 3 anddescribed above, and stores the picture in the acquired data DB 223.

(3-1-2) Line-of-Sight Detection Unit

The line-of-sight detection unit 221 calculates the line-of-sight originP and a line-of-sight vector 150 a indicating the direction of theline-of-sight 150 based on the pictures of the face, the eyeballs, theiris, and so on detected by the line-of-sight detection equipment 206.Hereafter, the direction of the line of sight is referred to as aline-of-sight vector in the present embodiment.

FIG. 9 is an explanatory view of an example of the method of calculatingthe line-of-sight origin P and the line-of-sight vector. For example,the feature points of the face are calculated based on the pictures ofthe face, the eyeballs, the iris, and so on, and compared with theamount of features of the face of the driver stored in advance. Next,the line-of-sight detection unit 221 extracts the direction of the facebased on the comparison result, the pictures of the face, the eyeballs,the iris, and so on, and detects the central position between a lefteyeball 152L and a right eyeball 152R illustrated in part (a) of FIG. 9as the line-of-sight origin P. Furthermore, the line-of-sight detectionunit 221 calculates the central position of an iris 153 a, that is, thecentral position of a pupil 153 b. Finally, the line-of-sight detectionunit 221 calculates the line-of-sight vector 150 a based on the centralposition between the line-of-sight origin P and the pupil 153 b. Sincethe driver may move his or her head forward, backward, leftward,rightward, up and down, the position of the line-of-sight origin Pcorresponding to the center point O of the spatial coordinate system ischanged depending on the position, the direction, and so on of the head.

The line-of-sight vector 150 a may be defined by the elevation angle θβmade by the line-of-sight vector 150 a and the XY plane and the azimuthθα which is made by the line-of-sight vector 150 a and the YZ plane asillustrated by parts (b) and (c) of FIG. 9. The line-of-sight vector 150a may also be defined by the coordinates in the spatial coordinatesystem with any center point O of the driver's vehicle 300.

The line-of-sight detection unit 221 stores the line-of-sight origin Pand the line-of-sight vector 150 a in the acquired data DB 223.

(3-1-3) Acquired Data DB 223

The acquired data DB 223 stores the peripheral information, theline-of-sight data detected by the line-of-sight detection unit 221, andso on. Furthermore, the acquired data DB 223 stores all necessary datasuch as the model of the driver's vehicle 300 and so on for the drivingpicture processing device 100 to generate display pictures. The acquireddata DB 223 may be configured by, for example, the RAM 203, the HDD 209a, an external recording medium, and so on.

(3-1-4) Transmission/Reception Unit

A transmission/reception unit 224 of the information acquisition device200 transmits and receives various types of data such as commands,peripheral information, line-of-sight data, the model of the driver'svehicle 300, and so on to and from the transmission/reception unit 120of the driving picture processing device 100.

(3-2) Driving Picture Processing Device

The driving picture processing device 100 functions as each functionalunit described later by executing a program with each hardwareconfiguration cooperating with others.

The driving picture processing device 100 according to the presentembodiment extracts the line-of-sight picture corresponding to theline-of-sight origin P and the line-of-sight vector from the peripheralpicture of the driver's vehicle 300. The line-of-sight picture includesa car window picture through a car window and/or a mirror picturethrough a mirror. The driving picture processing device 100 generates adisplay picture by removing the car body area of the driver's vehicle300 which cuts off the line of sight of the driver from theline-of-sight picture or superposing the car body picture such as apillar and so on which cuts off the line of sight of the driver on theline-of-sight picture.

The functional unit of the driving picture processing device 100includes, for example, the transmission/reception unit 120, a car windowpicture generation unit 121, a mirror picture generation unit 122, acutoff information calculation unit 123, a line-of-sight processing unit124, a display picture generation unit 125, and so on. Furthermore, tostore various types of information, the driving picture processingdevice 100 includes a peripheral information DB 126, a window picture DB127, a mirror picture DB 128, a cutoff information DB 129, line-of-sightdata DB 130, various correspondence table DBs 131, and so on. Each DBmay be configured by, for example, the RAM 103, the HDD 110 a, anexternal recording medium, and so on.

(3-2-1) Transmission/Reception Unit

The transmission/reception unit 120 of the driving picture processingdevice 100 transmits and receives various types of data, commands, andso on to and from the transmission/reception unit 222 of the informationacquisition device 200. The transmission/reception unit 120 acquiresvarious types of information such as the peripheral picture, theline-of-sight data, and so on acquired by the information acquisitiondevice 200 from the acquired data DB 223 through thetransmission/reception unit 222 in real time, and takes into each DB inthe driving picture processing device 100. In this case, thetransmission/reception unit 120 may acquire in real time various typesof information from the peripheral information acquisition unit 220 andthe line-of-sight detection unit 221 without the acquired data DB 223.Otherwise, the transmission/reception unit 222 may temporarily storevarious types of information about a series of operations in theacquired data DB 223 of the information acquisition device 200, andacquire the information later. That is, the various types of informationis not acquired in real time, but the various types of informationrelating to a series of operations is temporarily stored in the acquireddata DB 223, and the transmission/reception unit 222 acquires thevarious types of information from the acquired data DB 223 aftercompleting the series of operations.

The transmission/reception unit 120 may include a picture acquisitionunit for acquiring a peripheral picture and a line-of-sight acquisitionunit for acquiring line-of-sight data.

(3-2-2) Peripheral Information DB

The peripheral information DB 126 acquires the peripheral picture aroundthe driver's vehicle from the information acquisition device 200 as theperipheral information around the driver's vehicle and stores theinformation. The peripheral picture includes the pictures shot by theperipheral information acquisition equipment 205 configured by theforward camera 205 a, the right camera 205 b, the left camera 205 c, andthe backward camera 205 d.

FIG. 10 is an example of a peripheral information DB. The peripheralinformation DB 126 stores for each frame, for example, a frame numberand the picture data in each camera 205. The picture data includes aview ahead of the vehicle shot by the forward camera 205 a, a right sideview shot by the right camera 205 b, a left side view shot by the leftcamera 205 c, and a view behind the vehicle shot by the backward camera205 d.

(3-2-3) Line-of-Sight Data DB

The line-of-sight data DB 130 acquires the line-of-sight origin P andthe line-of-sight vector 150 a of the driver of the driver's vehiclefrom the information acquisition device 200 and stores the origin andvector.

FIG. 11 is an example of a line-of-sight data DB. The line-of-sight dataDB 130 acquires for each frame a frame number, the line-of-sight originP, and the line-of-sight vector 150 a from the information acquisitiondevice 200 and stores them. The line-of-sight vector 150 a is defined bythe azimuth θα and the elevation angle θ. The line-of-sight origin P maybe defined by the coordinates in the spatial coordinate system having acenter point O of the driver's vehicle 300 as an origin.

The line-of-sight data DB 130 further stores for each frame, as theinformation calculated by the line-of-sight processing unit 124described later, the effective vision range, whether or not there is amirror in the effective vision range, which mirror exists in theeffective vision range, and so on. The effective vision range refers tothe range in which the driver may view the line-of-sight origin P andthe line-of-sight vector.

(3-2-4) Various Correspondence Table DBs

The correspondence stored in each correspondence table DB 131 isdescribed below with the explanation of the method of generating adisplay picture.

The driving picture processing device 100 projects the peripheralpicture of the driver's vehicle 300 on a 3-dimensional projectionsurface 400, and generates a car window picture and a mirror picturecorresponding to the line-of-sight origin P and the line-of-sight vectorof the driver from the peripheral picture projected on the 3-dimensionalprojection surface 400. FIG. 12 is an explanatory view of the positionalrelationship between the peripheral picture projected on the3-dimensional projection surface and the car window picture and themirror picture corresponding to the line-of-sight origin P and theline-of-sight vector.

(a) Correspondence Between the Coordinates of Each Pixel of Each Cameraand the Coordinates of a 3-Dimensional Projection Surface

First, the picture data shot by each camera is processed and combined,thereby generating a peripheral picture projected on the 3-dimensionalprojection surface 400 as illustrated in FIG. 12. The 3-dimensionalprojection surface 400 is, for example, a bowl-shaped projection surfacehaving the driver's vehicle 300 at the center. Each correspondence tableDB 131 stores the correspondence between the coordinates of each pixelof each of the cameras 205 a through 205 d and the coordinates of the3-dimensional projection surface 400. Therefore, the car window picturegeneration unit 121 and the mirror picture generation unit 122 describedlater coordinate-convert the picture data of each pixel acquired by eachof the cameras 205 a through 205 d into the 3-dimensional projectionsurface 400 based on the correspondence above, and generate a peripheralpicture projected on the 3-dimensional projection surface 400.

(b) Correspondence Between the Line-of-Sight Origin P and theLine-of-Sight Vector, and the Car Window Vision Area

Described next is the correspondence between the line-of-sight origin Pand the line-of-sight vector, and the car window vision area. Asillustrated in FIG. 12, the position of the picture which may beobserved by the driver in the direction indicated by the line-of-sightvector 150 a from the line-of-sight origin P is associated with theposition of the peripheral picture of the 3-dimensional projectionsurface 400.

For example, the car window line-of-sight direction indicated by theline-of-sight vector 150 a 1 from the line-of-sight origin P is theforward direction from the driver's vehicle 300 through the car window.As a forward view indicated by the line-of-sight vector 150 a 1 from theline-of-sight origin P, a front window picture 400F is associated in theperipheral pictures of the 3-dimensional projection surface 400. Inaddition, the line-of-sight vector 150 a 1 extended from theline-of-sight origin P crosses the 3-dimensional projection surface 400at the intersection SPa. The intersection SPa corresponds to the end ofthe line of sight of the driver, and corresponds to the center of thefront window picture 400F. The car window line-of-sight is acquired bydirectly viewing an object through the car window and/or the portioncorresponding to the car window, and the car window line-of-sightdirection refers to the direction specified by the car windowline-of-sight. On the other hand, the mirror line-of-sight describedlater is an indirect line of sight through a mirror, and is acquiredafter the line-of-sight vector 150 a is reflected by the mirror 303.

Similarly, the car window line-of-sight direction indicated by theline-of-sight vector 150 a 2 from the line-of-sight origin P is thediagonally right forward direction of the driver's vehicle 300 throughthe car window. Furthermore, a right car window picture 400R in theperipheral pictures of the 3-dimensional projection surface 400 isassociated as a picture in the diagonally right forward directionindicated by the line-of-sight vector 150 a 2 from the line-of-sightorigin P. In addition, an intersection SPb at which the line-of-sightvector 150 a 2 extending from the line-of-sight origin P crosses the3-dimensional projection surface 400 corresponds to the center of theright car window picture 400R.

Similarly, the car window line-of-sight direction indicated by theline-of-sight vector 150 a 3 from the line-of-sight origin P is thediagonally left forward direction of the driver's vehicle 300 throughthe car window. Furthermore, a left car window picture 400L in theperipheral pictures of the 3-dimensional projection surface 400 isassociated as a picture in the diagonally left forward directionindicated by the line-of-sight vector 150 a 3 from the line-of-sightorigin P. In addition, an intersection SPc at which the line-of-sightvector 150 a 3 extending from the line-of-sight origin P crosses the3-dimensional projection surface 400 corresponds to the center of theleft car window picture 400L.

Thus, the range of the car window vision area which may be observed bythe driver on the 3-dimensional projection surface 400 through a windowdepends on the car window line-of-sight direction indicated by theline-of-sight origin P and the line-of-sight vector 150 a. Eachcorrespondence table DB 131 stores the correspondence between theline-of-sight origin P and the line-of-sight vector 150 a, and the carwindow vision area on the 3-dimensional projection surface 400. The carwindow vision area is a vision area of the driver on the 3-dimensionalprojection surface 400 when an object is viewed through a car window anda portion corresponding to the car window, and is a vision area when thecar window line-of-sight of the driver is not cut off by the car body ofthe driver's vehicle 300. If the car window cutoff information about thecar body is added using the pillar described later and so on to the carwindow picture corresponding to the car window vision area, then the carwindow display picture with the car window line-of-sight of the drivercut off by the car body may be generated.

FIG. 18 is an example of the association between the line-of-sightorigin p and the line-of-sight vector, and the car window vision area onthe 3-dimensional projection surface. Each correspondence table DB 131stores each line-of-sight origin P and the line-of-sight vectorassociated with the car window vision area on the 3-dimensionalprojection surface. For example, in the case of the line-of-sight originP=(X1, Y1, Z1), and the line-of-sight vector=(θαa, θβa), the car windowvision area 1 a is associated. The car window vision area may beexpressed by the information indicating the range of a coordinate groupin the case of the coordinates on the 3-dimensional projection surface400 in, for example, the spatial coordinate system having the centerpoint O of the driver's vehicle 300 as an origin.

As illustrated in FIG. 13, the correspondence table DB 131 may storeeach line-of-sight origin P and line-of-sight vector as associated withthe intersection SP with the 3-dimensional projection surface.

(c) Correspondence Between the Line-of-Sight Vector from theLine-of-Sight Origin P with the Car Window Cutoff Information

Next, when the periphery is observed from the window of the driver'svehicle 300, the car window line-of-sight of the driver is cut off bythe car body such as the pillar of the driver's vehicle 300. The area inwhich the car window line-of-sight of the driver depends on theline-of-sight origin P and the line-of-sight vector 150 a. For example,when the line-of-sight vector of the driver refers to the diagonallyright forward direction, for example, the right pillar 307R is locatedat the center of the vision of the driver. On the other hand, forexample, when the line-of-sight vector of the driver refers to thediagonally left forward direction, the left pillar 307L is located atthe center of the vision of the driver.

Each correspondence table DB 131 stores the line-of-sight origin P andthe line-of-sight vector 150 a as associated with the car window cutoffinformation about the car body of the driver's vehicle 300 which cutsoff the car window line-of-sight of the driver. The car window cutoffinformation is the information about the cut off of the line of sight ofthe driver in the range of the car window vision area associated withthe line-of-sight origin P and the line-of-sight vector 150 a. The carwindow cutoff information also includes the car body area and/or carbody picture of the driver's vehicle 300 which cuts off the car windowline-of-sight of the driver. The car body area may be expressed by theinformation about the range of a coordinate group in the case of, forexample, the coordinates in the display area of the display 255.Furthermore, the car body picture may be configured by thecorrespondence between the picture data as displayed in the display areaof the display 255 and the coordinates on the display, and so on. Thecar body picture includes the pictures of the front pillar 307F, thedashboard 301, the right pillar 307R, the left pillar 307L, and so on.

FIG. 14 is an example of the correspondence between the line-of-sightorigin P and the line-of-sight vector, and the car window cutoffinformation. Since the car body structure is different for each model,each correspondence table DB 131 stores for each model the line-of-sightorigin P and the line-of-sight vector associated with the car windowcutoff information. For example, in the case of the model=A, theline-of-sight origin P=(X1, Y1, Z1), the line-of-sight vector=(θαa,θβa), the car window cutoff information A1a is associated.

(d) Correspondence Between the Line-of-Sight Origin P and the MirrorVision Area

When the mirror 303 exists in the effective vision range with respect tothe line-of-sight origin P and the line-of-sight vector of the driver,the driver may visually confirm the backward and diagonally backwardconditions of the driver's vehicle 300. Each correspondence table DB 131stores the mirror information such as the position of a mirror etc. andthe line-of-sight origin P as associated with the mirror vision areawhich may be visually confirmed by the driver from the line-of-sightorigin P through the mirror 303.

(d-1) Effective Vision Range

Described first is the relationship between the effective vision rangeand the mirror. The driving picture processing device 100 according tothe present embodiment displays the mirror display picture through themirror 303 on the display 255 if any mirror 303 exists in the effectivevision range. The effective vision range is visually confirmed withrespect to the line of sight of the driver, and is defined by, forexample, the effective vision angle θe having as the center thedirection indicated by the line-of-sight vector 150 a from theline-of-sight origin P. The effective vision range may be also definedby a set of coordinates of the spatial coordinate system having as theorigin the center point O of the driver's vehicle 300.

FIG. 15 is an explanatory view of the relationship between the effectivevision range and the mirror, and the visual mirror confirmation rangewhich may be visually confirmed through the mirror, the parts (a)through (c) in FIG. 15 are examples of the case in which the mirror 303exists in the effective vision range.

In the part (a) in FIG. 15, the line-of-sight 150 of the driver isdirected toward the left door mirror 303L, and the driver directly looksat the left door mirror 303L. That is, the left door mirror 303L islocated at the central part of the effective vision range. In this case,the line-of-sight 150 of the driver is reflected by the left door mirror303L, thereby indicating a mirror line-of-sight 155. That is, theline-of-sight vector 150 a from the line-of-sight origin P is reflectedby the left door mirror 303L, thereby indicating a mirror line-of-sightvector 155 a. The left door mirror 303L has a specified shape and area,and the driver may visually confirm the state and so on of a specifiedvisual mirror confirmation range in the backward and diagonally backwardconditions of the driver's vehicle 300 through the left door mirror303L.

In the part (b) in FIG. 15, the line-of-sight vector 150 a of the driveris directed to the body of the driver, and is not directed in thedirection of the mirror 303. However, the right door mirror 303R and theleft door mirror 303L are located in the effective vision range definedby the effective vision angle θe1 having the line-of-sight vector 150 aat the center. Therefore, the driving picture processing device 100estimates that the driver may visually confirm the condition and so onof the specified visual mirror confirmation range in the backward anddiagonally backward conditions of the driver's vehicle 300 through theright door mirror 303R and the left door mirror 303L. In this case, theeffective vision angle θe1 in the case of the part (b) in FIG. 15 isexpressed by the angle on the XY surface as a horizontal plane.

The effective vision angle θe may be defined not only by the angle θe1on the XY surface, but also by the angle made with the XY surface. Inthe part (c) in FIG. 15, the line-of-sight vector 150 a of the driverhas a specified angle with respect to the XY surface, and the backmirror 303B is located in the effective vision range defined by theeffective vision angle θe2 having the line-of-sight vector 150 a as thecenter. Therefore, the driving picture processing device 100 estimatesthat the driver may visually confirm through the back mirror 303B thestate and so on of a specified visual mirror confirmation range in thebackward or diagonally backward conditions of the driver's vehicle 300.

Although the mirror 303 exists in the range of the effective visionangle θe1 in the XY plane with respect to the line-of-sight vector 150a, there is the case in which the mirror 303 is not located in the rangeof the effective vision angle θe2 made with the XY plane. In this case,the driving picture processing device 100 may determine that the mirror303 is not visually confirmed. For example, assume that the back mirror303B is located in the effective vision angle θe1 with respect to theline-of-sight vector 150 a, but is not located in the effective visionangle θe2. In this case, the driving picture processing device 100determines that the line of sight of the driver is directed downward andthat the driver does not visually confirm the back mirror 303B.

(d-2) Visual Mirror Confirmation Range

Described below is the visual mirror confirmation range. Theline-of-sight processing unit 124 calculates the virtual line-of-sightorigin VP and the mirror vision field angle θm according to the mirrorinformation including the model, the mirror position, the mirror angle,the shape of the mirror, and so on. The visual mirror confirmation rangeis determined by the virtual line-of-sight origin VP, the mirror visionfield angle θm, and so on. The virtual line-of-sight origin VP is anorigin for determination of the visual mirror confirmation range inwhich the driver may visually confirm an object through the mirror 303.The mirror vision field angle θm is an angle for definition of thevisual mirror confirmation range using the virtual line-of-sight originVP as an origin.

For example, in the case in part (a) in FIG. 15, the left door mirror303L is in the effective vision range, and the visual mirrorconfirmation range is defined by the mirror vision field angle θmL madewith the mirror line-of-sight vector 155 a 1 and the mirrorline-of-sight vector 155 a 2 using the virtual line-of-sight origin VPas an origin. The mirror line-of-sight vectors 155 a 1 and 155 a 2 arethe endmost vectors in the visually confirmable range by the driverthrough the left door mirror 303L, and are vectors on the boundary withthe range in which an object is not visually confirmed. In addition, forexample, in the case of part (b) in FIG. 15, the right door mirror 303Rand the left door mirror 303L are in the effective vision range. In thiscase, the visual mirror confirmation range includes the visual mirrorconfirmation range by the right door mirror 303R and the visual mirrorconfirmation range by the left door mirror 303L. The visual mirrorconfirmation range by the left door mirror 303L is similar to the casein part (a) in FIG. 15. The visual mirror confirmation range by theright door mirror 303R is defined by the mirror vision field angle θmRmade with the mirror line-of-sight vector 155 a 3 and the mirrorline-of-sight vector 155 a 4 using the virtual line-of-sight origin VPas an origin. The mirror line-of-sight vectors 155 a 3 and 155 a 4 areendmost vectors in the visually confirmable range through the right doormirror 303R.

Each correspondence table DB 131 stores the mirror information and theline-of-sight origin P as associated with the virtual line-of-sightorigin VP and the mirror vision field angle θm. The mirror picturegeneration unit 122 calculates the virtual line-of-sight origin VP andthe mirror vision field angle θm based on the correspondence above, andmay calculate the visual mirror confirmation range. FIGS. 16A and 16Bare an example of the correspondence between the mirror information andthe line-of-sight origin P, and the virtual line-of-sight origin VP, themirror vision field angle θm, the mirror vision area, and the mirrorcutoff information for each model. In FIGS. 16A and 16B, as an example,the mirror information is defined by the position of the mirror 303, andthe angle defined by the azimuth θ_(Y) and the elevation angle θδindicating the attachment angle of the mirror. For example, with themodel=A, the mirror position=(Xm1, Ym1, Zm1), the mirror angle=(θ_(Y)a,θδa), and the line-of-sight origin P=(X1, Y1, Z1), associated are thevirtual line-of-sight origin VP=(XA1a, YA1a, ZA1a) and the mirrorangle=θmA1a.

(d-3) Mirror Vision Area

Described next is the mirror vision area. The mirror picture generationunit 122 described later calculates the virtual line-of-sight origin VPand the mirror vision field angle θm based on the mirror information andthe line-of-sight origin P, based on which the mirror vision area on the3-dimensional projection surface 400 is calculated.

However, although the mirror vision area may be calculated in theprocess above, the mirror vision area may be calculated based on thecorrespondence between the mirror information and the line-of-sightorigin P for each model, and the each mirror vision area on the3-dimensional projection surface 400.

The mirror vision area is a vision area of the driver on the3-dimensional projection surface 400 through the mirror 303, and is avision area when the mirror line-of-sight 155 reflected by the mirror303 is not cut off by the car body of the driver's vehicle 300. When themirror cutoff information about the car body by the pillar describedlater is added to the mirror picture corresponding to the mirror visionarea, the mirror display picture when the line of sight of the driver iscut off by the car body is generated.

Each mirror vision area which may be visually confirmed by each of themirrors 303R, 303L, and 303B is described using FIG. 12 again.

For example, assume that there is the right door mirror 303R in theeffective vision range defined by the direction indicated by theline-of-sight vector 150 a from the line-of-sight origin P. In thiscase, a right mirror picture 400MR is associated in the peripheralpictures of the 3-dimensional projection surface 400 as the picture inthe visual mirror confirmation range through the right door mirror 303R.Similarly, when there is the left door mirror 303L in the effectivevision range, a left side view 400ML is associated in the peripheralpictures of the 3-dimensional projection surface 400 as the picture inthe visual mirror confirmation range through the left door mirror 303L.Similarly, when the back mirror 303B is located in the effective visionrange, a back mirror picture 400MB in the peripheral pictures of the3-dimensional projection surface 400 is associated as the picture of thevisual mirror confirmation range through the back mirror 303B.

Thus, the mirror vision area which may be observed by the driver throughthe mirror 303 in the 3-dimensional projection surface 400 depends onthe mirror 303 located in the effective vision range. Eachcorrespondence table DB 131 stores the correspondence between the mirrorinformation and the line-of-sight origin P, and the each mirror visionarea on the 3-dimensional projection surface 400 for each model asillustrated in FIGS. 16A and 16B. For example, assume that the model=A,the mirror position=(Xm1, Ym1, Zm1), the mirror angle=(θ_(Y) a, θδa),and the line-of-sight origin P=(X1, Y1, Z1). In this case, the backmirror 303B, the right door mirror 303R, and the left door mirror 303Lare associated with the back mirror vision area A1 a, the right doormirror vision area A1 a, and the left door mirror vision area A1 a. Themirror vision area may be expressed by the information about the set ofthe coordinates, the range of a coordinate group on the 3-dimensionalprojection surface 400 in the spatial coordinate system using the centerpoint O of the driver's vehicle 300 as an origin.

The line-of-sight processing unit 124 designates the mirror 303 in theeffective vision range from the line-of-sight data DB 130 based on theline-of-sight origin P and the line-of-sight vector. Furthermore, themirror picture generation unit mirror picture generation unit 122 readsthe mirror vision area of the mirror 303 in the effective vision rangein the three mirror vision areas corresponding to the line-of-sightorigin P, thereby generating a mirror picture.

(e) Correspondence Between the Line-of-Sight Origin P and the MirrorCutoff Information

When the driver observes the periphery through the mirror 303 of thedriver's vehicle 300, the mirror line-of-sight 155 reflected by themirror 303 or the driver is cut off by the car body of a pillar and soon of the driver's vehicle 300. In addition, by the reflection by thewindow may cut off the mirror line-of-sight 155 of the driver.

Each correspondence table DB 131 stores the mirror information and theline-of-sight origin P as associated with the mirror cutoff informationabout the car body of the driver's vehicle 300 which cuts off the mirrorline-of-sight 155 of the driver for each model. The mirror cutoffinformation includes the area of the car body and/or the car bodypicture of the driver's vehicle 300 which cut off the mirrorline-of-sight 155 of the driver. For example, with the model=A, themirror position=(Xm1, Ym1, Zm1), the mirror angle=(θ_(Y)a, θδa), and theline-of-sight origin P=(X1, Y1, Z1), the back mirror cutoff informationAla, the right mirror cutoff information Ala, and the left mirror cutoffinformation Ala are associated.

(f) Position of Mirror Display Area in Display Area of Display

Next, the position of the mirror display area 266 corresponding to thedisplay area of the display 255 is described with reference to FIGS. 12,and 17 through 19. FIGS. 17 through 19 are explanatory views of therelationship between the car window picture on the 3-dimensionalprojection surface and the mirror picture, and the display area of thedisplay.

The driving picture processing device 100 generates a car window pictureand/or a mirror picture from the peripheral picture on the 3-dimensionalprojection surface 400 based on the line-of-sight origin P and theline-of-sight vector 150 a. Furthermore, the driving picture processingdevice 100 generates a display picture obtained by adding the car windowcutoff information and/or mirror cutoff information to the car windowpicture and/or mirror picture. The display area of the display 255includes the car window display area 265 and the mirror display area266. The mirror display area 266 is a part of the areas of the displayareas of the display 255, and the car window display area 265 is adisplay area of the display 255 excluding the mirror display area 266.The car window display area 265 displays a car window display picturemade of the car window picture and the car window cutoff information.The mirror display area 266 displays a mirror display picture made ofthe mirror picture and mirror cutoff information. If the line-of-sightorigin P and the line-of-sight vector 150 a change, the position of themirror 303 in the vision of the driver also changes. Therefore, theposition of the mirror display area 266 in the display area of thedisplay 255 also changes.

For example, in FIG. 12, assume that the driver looks ahead, theline-of-sight data of the driver is the line-of-sight origin P and theline-of-sight vector 150 a 1, and the there is the back mirror 303B inthe effective vision range. The driver may visually confirm the frontwindow picture 400F and the back mirror picture 400MB. In this case, asillustrated in FIG. 17, the front window picture 400F is displayed inthe car window display area 265 in the display area of the display 255.In addition, the back mirror picture 400MB is displayed in the backmirror display area 266B in the display area of the display 255. In thiscase, the intersection SPa between the line of sight of the driver andthe 3-dimensional projection surface 400 is coordinate-converted intothe intersection SPa′ at the center of the display area of the display255. The point MPa of the back mirror picture 400MB iscoordinate-converted into the point MPa′ of the back mirror display area266B.

In FIG. 12, assume that the driver is headed in the diagonally rightforward direction, the line-of-sight data of the driver is theline-of-sight origin P and the line-of-sight vector 150 a 2, and thereare the back mirror 303B and the right door mirror 303R in the effectivevision range. The driver may visually confirm the right car windowpicture 400R, the back mirror picture 400MB, and the right mirrorpicture 400MR. In this case, as illustrated in FIG. 18, the right carwindow picture 400R is displayed in the car window display area 265. Inaddition, the back mirror picture 400MB is displayed in the back mirrordisplay area 266B in the display area of the display 255. Furthermore,the right mirror picture 400MR is displayed in the right mirror displayarea 266R. In this case, the intersection SPb between the line of sightof the driver and the 3-dimensional projection surface 400 iscoordinate-converted to the point SPb′ at the central portion of thedisplay area of the display 255. The point MPa of the back mirrorpicture 400MB is coordinate-converted to the point MPa′ of the backmirror display area 266B. Furthermore, the point MPb of the right mirrorpicture 400MR is coordinate-converted to the point MPb′ of the rightmirror display area 266R.

Furthermore, in FIG. 12, assume that the driver is headed in thediagonally left forward direction, the line-of-sight data of the driveris the line-of-sight origin P and the line-of-sight vector 150 a 3, andthere are the back mirror 303B and the left door mirror 303L in theeffective vision range. The driver may visually confirm the left carwindow picture 400L, the back mirror picture 400MB, and the left mirrorpicture 400ML. In this case, as illustrated in FIG. 19, the left carwindow picture 400L is displayed in the car window display area 265. Inaddition, the back mirror picture 400MB is displayed in the back mirrordisplay area 266B, and the left mirror picture 400ML is displayed on theleft mirror display area 266L. In this case, the intersection SPcbetween the line of sight of the driver and the 3-dimensional projectionsurface 400 is coordinate-converted to the point SPc′ at the centralportion of the display area of the display 255. The point MPa of theback mirror picture 400MB is coordinate-converted to the point MPa′ ofthe back mirror display area 266B. Furthermore, the point MPc of theleft mirror picture 400ML is coordinate-converted to the point MPc′ ofthe right mirror display area 266L.

Thus, the position of the mirror display area 266 in the display area ofthe display 255 depends on the line-of-sight origin P and theline-of-sight vector. Each correspondence table DB 131 stores theline-of-sight origin P and the line-of-sight vector as associated witheach mirror display area as illustrated in FIG. 20. FIG. 20 is anexample of the correspondence between the line-of-sight origin P and theline-of-sight vector, and each mirror display area. For example, withthe line-of-sight origin P=(X1, Y1, Z1), and the line-of-sightvector=(θαa, θβa), the back mirror display area 266B and the rightmirror display area 266R are associated.

(g) Others

Each correspondence table DB 131 stores all other information about themodel of the vehicle whose display picture for generation of the displaypicture by the driving picture processing device 100, the angle of theeffective vision angle θe, and so on. The effective vision angle θe isset as, for example, a vision angle which may visually confirmed by acommon driver.

In addition, the correspondence of each correspondence table DB 131 isperformed by considering the distortion correction performed when apicture taken by a camera is projected on the 3-dimensional projectionsurface 400, the distortion correction performed when the peripheralpicture projected on the 3-dimensional projection surface 400 isconverted on the display 255, and so on.

Each correspondence table DB 131 may be stored with the above-mentionedcorrespondence using, for example, an equation. For example, therelationship between the line-of-sight origin P and the line-of-sightvector in FIG. 13 and the car window vision area on the 3-dimensionalprojection surface 400 may be regulated by an equation and then stored.

The above-mentioned correspondence is only an example, and, for example,a more detailed correspondence may be performed, and a roughercorrespondence may be presented.

T (3-2-5) Line-of-Sight Picture Generation Unit

The line-of-sight processing unit 124 calculates the effective visionrange, and determines whether or not there is the mirror 303 in theeffective vision range as illustrated in

FIG. 15.

The line-of-sight processing unit 124 reads the line-of-sight origin Pand the line-of-sight vector 150 a from the line-of-sight data DB 130,and calculates the effective vision range based on the line-of-sightorigin P, the line-of-sight vector 150 a, and the effective vision angleθe as a specified angle. The effective vision range is defined by theeffective vision angle θe using as the center the line-of-sight vector150 a extending from the line-of-sight origin P, and is defined by a setof coordinates of the spatial coordinate system.

Next, the line-of-sight processing unit 124 determines which mirror 303exists in the effective vision range as illustrated in parts (b) and (c)in FIG. 15 based on the mirror position of each mirror 303 of thedriver's vehicle 300. For example, when the coordinates indicating themirror position of the left door mirror 303L are included in the set ofcoordinates which define the effective vision range, the line-of-sightprocessing unit 124 determines that the left door mirror 303L isincluded in the effective vision range.

The line-of-sight processing unit 124 stores the effective vision rangeand the determination result in the line-of-sight data DB 130. Theline-of-sight data DB 130 stores, as illustrated in FIG. 11, theeffective vision range in each frame, the type of mirror existing in theeffective vision range, and “NO” when no mirror exists in the effectivevision range. For example, in the frame of the frame number 3, theline-of-sight origin P=(XP3, YP3, ZP3), the line-of-sightvector=(θα_(—)3, θβ_(—)3) are stored. In this case, the effective visionrange=range_(—)3, and the back mirror 303B and the right door mirror303R exist in the effective vision range. On the other hand, in theframe of the frame number 4, the line-of-sight origin P=(XP4, YP4, ZP4),and the line-of-sight vector=(θα_(—)4, θβ_(—)4) are stored, but there isno mirror 303 in the effective vision range=range_(—)4. Therefore, “NO”is stored.

(3-2-6) Car Window Picture Generation Unit, Car Window Picture DB

The car window picture generation unit 121 generates a car windowpicture corresponding to the line-of-sight origin P of the driver andthe line-of-sight vector from the peripheral picture of the driver'svehicle 300.

For example, the car window picture generation unit 121 reads theperipheral information about the target frame from the peripheralinformation DB 126 in FIG. 10, and projects the information on the3-dimensional projection surface 400 as illustrated in FIG. 12. The carwindow picture generation unit 121 reads the line-of-sight origin P andthe line-of-sight vector 150 a from the line-of-sight data DB 130 inFIG. 11 relating to the target frame. Next, the car window picturegeneration unit 121 reads the car window vision area on the3-dimensional projection surface 400 from the correspondence table DB131 in FIG. 13 based on the line-of-sight origin P and the line-of-sightvector 150 a. Finally, the car window picture generation unit 121extracts the picture corresponding to the car window vision area, fromthe 3-dimensional projection surface 400 on which the peripheral pictureof the driver's vehicle 300 is projected, and processes the picture intothe car window picture which may be displayed in the car window displayarea 265 of the display 255.

The window picture DB 127 stores the car window picture generated by thecar window picture generation unit 121. FIG. 21 is an example of awindow picture DB. The window picture DB 127 stores a car window picturefor each frame.

(3-2-7) Mirror Picture Generation Unit, Mirror Picture DB

The mirror picture generation unit 122 generates a mirror picture whichmay be visually confirmed by the mirror 303 when there is any mirror 303in the effective vision range in the target frame.

For example, as with the car window picture generation unit 121, themirror picture generation unit 122 projects the peripheral informationabout the target frame on the 3-dimensional projection surface 400.Otherwise, the mirror picture generation unit 122 may use the peripheralpicture of the 3-dimensional projection surface 400 generated by the carwindow picture generation unit 121.

The mirror picture generation unit 122 reads the line-of-sight origin Pand the information about which mirror 303 exists in the effectivevision range relating to a target frame from the line-of-sight data DB130 in FIG. 11. Next, the mirror picture generation unit 122 reads themirror vision area of the corresponding mirror from each correspondencetable DB 131 based on the line-of-sight origin P and the mirror 303 inthe effective vision range. For example, assume that the mirror picturegeneration unit 122 determines that there is the back mirror 303B andthe right door mirror 303R in the effective vision range by referring tothe line-of-sight data DB 130 relating to certain line-of-sight origin Pand line-of-sight vector. In this case, the mirror picture generationunit 122 refers to each correspondence table DB 131 in FIGS. 16 A and16B, and reads the back mirror vision area and the right vision area inthe three mirror vision areas associated with the correspondingline-of-sight origin P.

Finally, the mirror picture generation unit 122 extracts each picturecorresponding to each mirror vision area from the 3-dimensionalprojection surface 400 on which the peripheral picture of the driver'svehicle 300 is projected, and processes each picture into a mirrorpicture which may be displayed in the mirror display area 266 of thedisplay 255.

The mirror picture generation unit 122 refers to the line-of-sight dataDB 130 in FIG. 11, and when it determines that there is no mirror 303 inthe effective vision range, no mirror picture is generated.

The mirror picture DB 128 stores the mirror picture generated by themirror picture generation unit 122. FIG. 22 is an example of a mirrorpicture DB. The mirror picture DB 128 stores the type of mirror 303 inthe effective vision range, and the mirror picture for each frame. Whenthere are a plurality of mirrors 303 in the effective vision range, itstores each mirror picture of each mirror in one frame. In addition,when there is no mirror 303 in the effective vision range, “NO” isstored.

(3-2-8) Cutoff Information Generation Unit, Cutoff Information DB

The cutoff information calculation unit 123 generates cutoff informationabout the car body of the driver's vehicle 300 which cuts off the lineof sight of the driver. The cutoff information includes cutoffinformation about the car window which cuts off the car windowline-of-sight of the driver, and mirror cutoff information about the cutoff of the mirror line-of-sight of the driver reflected by the mirror303.

For example, the cutoff information calculation unit 123 reads theline-of-sight origin P and the line-of-sight vector 150 a from theline-of-sight data DB 130 in FIG. 11 relating to the target frame. Inaddition, the cutoff information calculation unit 123 reads the carwindow cutoff information from each correspondence table DB 131 in FIG.14 based on the type of the driver's vehicle, the line-of-sight originP, and the line-of-sight vector 150 a, and stores the information in thecutoff information DB 129.

Furthermore, the cutoff information calculation unit 123 reads theinformation about which mirror 303 exists in the effective vision rangefrom the line-of-sight data DB 130 in FIG. 11 relating to the targetframe. The cutoff information calculation unit 123 reads the mirrorcutoff information about the corresponding mirror 303 from eachcorrespondence table DB 131 in FIGS. 16 A and 16B based on theline-of-sight origin P and the mirror 303 in the effective vision range,and stores the information in the cutoff information DB 129.

FIG. 23 is an example of the cutoff information DB. The cutoffinformation DB 129 stores for each frame the car window cutoffinformation, the type of the mirror 303 in the effective vision range,and the mirror cutoff information. When there are a plurality of mirrors303 in the effective vision range, one frame stores the mirror cutoffinformation about each mirror 303. If there is not mirror 303 in theeffective vision range, “NO” is stored.

(3-2-9) Display Picture Generation Unit

(a) Generating Car Window Display Picture

The display picture generation unit 125 generates a car window displaypicture based on the car window picture in the window picture DB 127 andthe car window cutoff information in the cutoff information DB 129 foreach frame. For example, in the case of the frame of the frame number 1,the display picture generation unit 125 reads the car windowpicture_(—)1 from the window picture DB 127 in FIG. 21. In addition, thedisplay picture generation unit 125 reads the car window cutoffinformation_(—)1 of the frame number 1 from the cutoff information DB129 in FIG. 23. The display picture generation unit 125 generates thecar window display picture_(—)1 of the frame number 1 based on thewindow picture_(—)1 and the car window cutoff information_(—)1. In thiscase, the display picture generation unit 125 generates a car windowdisplay picture by removing the car window cutoff information as a carbody area such as a pillar and so on which cuts off the line of sight ofthe driver from the car window picture, thereby generating a car windowdisplay picture. Otherwise, for example, the display picture generationunit 125 generates a car window display picture by superposing the carwindow cutoff information as a car body picture such as a pillar whichcuts off the line of sight of the driver toward the periphery of thedriver's vehicle on the car window picture.

FIG. 24 is an example of a picture used in a car window display picture.FIGS. 25 through 27 are examples of car window display pictures. Asillustrated in FIG. 24, the driver's vehicle 300 is traveling on thetraffic lane 600. Ahead of the driver's vehicle 300, another vehicle 500a is traveling on the traffic lane 600. Diagonally right ahead of thedriver's vehicle, a vehicle 500 b is travelling on a traffic lane 601. Awalker 500 c is walking on a sidewalk 602.

In the state in FIG. 24, as indicated by the line-of-sight origin P andthe line-of-sight vector 150 a 1 in FIG. 12, it is assumed that thedriver is looking ahead. In this case, the display picture generationunit 125 generates a car window picture as illustrated in part (a) inFIG. 25. In part (a) in FIG. 25, the pictures through car windowsincluding the vehicles 500 a and 500 b, and the walker 500 c aredisplayed. Furthermore, the car window cutoff information is combinedwith the pictures through the car windows in part (a) in FIG. 25,thereby generating a car window display picture illustrated by (b) inFIG. 25. In part (b) in FIG. 25, the car body area which cuts off theline of sight is removed from the car window picture, and the car windowdisplay picture is generated. The car body area which cuts off the lineof sight is indicated by diagonal lines which are not observed by thedriver. The car body area in part (b) in FIG. 25 includes, for example,a car body area 280F by the front pillar 307F, a car body area 280R bythe right pillar 307R, a car body area 280 by the right pillar 307R, acar body area 280D by a car body area 280L, and the dashboard 301. Thepoint SPa′ is the central part of the display area of the display 255.

In the state in FIG. 24, as indicated by the line-of-sight origin P andthe line-of-sight vector 150 a 2 in FIG. 12, it is assumed that thedriver is looking diagonally right ahead. In this case, the displaypicture generation unit 125 generates a car window picture asillustrated in part (a) in FIG. 26. In part (a) in FIG. 25, the picturesthrough car windows including the vehicles 500 a and 500 b aredisplayed. Furthermore, the car window cutoff information is combinedwith the pictures through the car windows in part (a) in FIG. 25,thereby generating a car window display picture illustrated by (b) inFIG. 25. In part (b) in FIG. 25, the car body area includes, forexample, the car body area 280F by the front pillar 307F, the car bodyarea 280R by the right pillar 307R, and the car body area 280D by thedashboard 301.

Furthermore, in the state in FIG. 24, as indicated by the line-of-sightorigin P and the line-of-sight vector 150 a 3 in FIG. 12, it is assumedthat the driver is looking diagonally left ahead. In this case, thedisplay picture generation unit 125 generates a car window picture asillustrated in part (a) in FIG. 27. In part (a) in FIG. 27, the picturesthrough car windows including the vehicle 500 a and walker 500 c aredisplayed. Furthermore, the car window cutoff information is combinedwith the pictures through the car windows in part (a) in FIG. 27,thereby generating a car window display picture illustrated by (b) inFIG. 27. In part (b) in FIG. 27, the car body area includes, forexample, the car body area 280F by the front pillar 307F, the car bodyarea 280L by the left pillar 307L, and the car body area 280D by thedashboard 301.

(b) Generating Mirror Display Picture

The display picture generation unit 125 generates a mirror displaypicture according to the mirror picture in the mirror picture DB 128 andthe mirror cutoff information about the cutoff information DB 129 whenthere is a mirror in the effective vision range. For example, withreference to FIG. 22, since there is no mirror in the effective visionrange in the frame of frame number 1, the display picture generationunit 125 generates no mirror display picture. On the other hand, thedisplay picture generation unit 125 reads the back mirror picture_(—)2from the mirror picture DB 128 in FIG. 22. Furthermore, the displaypicture generation unit 125 reads the mirror cutoff information_(—)2 ofthe frame number 2 from the cutoff information DB 129 in FIG. 23. Thedisplay picture generation unit 125 generates a mirror displaypicture_(—)2 with the frame number 2 according to the mirrorpicture_(—)2 and the mirror cutoff information_(—)2. The method ofgenerating a mirror display picture from the mirror picture and themirror cutoff information is the same as that of the above-mentioned carwindow display picture.

(c) Combining Car Window Display Picture and Mirror Display Picture

The display picture generation unit 125 combines the car window displaypicture with the mirror display picture, thereby generating a displaypicture.

For example, the display picture generation unit 125 reads theline-of-sight origin P and the line-of-sight vector from theline-of-sight data DB 130 in FIG. 11. Furthermore, the display picturegeneration unit 125 reads the mirror display area based on theline-of-sight origin P and the line-of-sight vector from eachcorrespondence table DB 131 in FIG. 20. Then, the display picturegeneration unit 125 superposes the mirror display picture on the carwindow display picture based on the mirror display area, therebygenerating a display picture.

FIG. 28 is an example of a display picture obtained by superposing aback mirror picture on the back mirror display area 266B in part (b) inFIG. 26. FIG. 29 is an example of a display picture obtained bysuperposing a right mirror picture on the right mirror display area 266Rin part (b) in FIG. 26. Thus, the display picture generation unit 125generates a display picture on which a mirror picture is superposed inthe mirror display area of the mirror 303 in the effective vision area.The display picture generation unit 125 does not superpose a mirrorpicture on the mirror display area of the mirror 303 not located in theeffective vision range.

(3-3) Drive Training Terminal

The functional unit of the drive training terminal 250 in FIG. 8includes, for example, a transmission/reception unit 270 and a displaycontrol unit 271. The drive training terminal 250 accepts an instructionto display a desired display picture from a viewer through the mouse 256and the keyboard 257. The transmission/reception unit 270 outputs theinstruction to display the display picture to the display picturegeneration unit 125 of the driving picture processing device 100. Thetransmission/reception unit 270 also receives a desired display picturefrom the display picture generation unit 125, and displays the displaypicture on the display 255.

(4) Flow of Processes

Described below is a flow of processes performed by the driving pictureprocessing device 100 according to the first embodiment.

FIG. 30 is a flowchart of an example of the flow of the processesperformed by the driving picture processing device according to thefirst embodiment. The driving picture processing device 100 acquiresperipheral information and line-of-sight data from the informationacquisition device 200 for each frame, and stores them in the peripheralinformation DB 126 and the line-of-sight data DB 130. The followingprocesses are performed on, for example, each frame.

Steps S1, S2: The driving picture processing device 100 sequentiallyadds the frame numbers i from 0.

Step S3: The car window picture generation unit 121 and the mirrorpicture generation unit 122 reads the peripheral information from theperipheral information DB 126, and reads the line-of-sight data from theline-of-sight data DB 130 for the frame number i. The cutoff informationcalculation unit 123, the line-of-sight processing unit 124, and thedisplay picture generation unit 125 reads the line-of-sight data fromthe line-of-sight data DB 130 for the target frame number i.

Step S4: The car window picture generation unit 121 projects theperipheral information about the target frame to the 3-dimensionalprojection surface 400.

Step S5: The car window picture generation unit 121 reads the car windowvision area on the 3-dimensional projection surface 400 from thecorrespondence table DB 131 based on the line-of-sight origin P and theline-of-sight vector 150 a. Next, the car window picture generation unit121 extracts a picture corresponding to the car window vision area fromthe 3-dimensional projection surface 400. Furthermore, the car windowpicture generation unit 121 processes the extracted picture into a carwindow picture which may be displayed in the car window display area 265of the display 255, and stores the picture in the window picture DB 127.

Step S6: The cutoff information calculation unit 123 reads the carwindow cutoff information from the correspondence table DB 131 based onthe type of the driver's vehicle, the line-of-sight origin P, and theline-of-sight vector 150 a from the correspondence table DB 131, andstores the information in the cutoff information DB 129.

Step S7: The display picture generation unit 125 reads the car windowpicture in the window picture DB 127 and the car window cutoffinformation in the cutoff information DB 129 for the target frame numberi, combines the car window picture with the car window cutoffinformation, and generates a car window display picture.

Step S8: The line-of-sight processing unit 124 calculates the effectivevision range based on the line-of-sight origin P and the line-of-sightvector 150 a, and the effective vision angle θe of a specified angle asillustrated in part (b) and (c) in FIG. 15. Next, the line-of-sightprocessing unit 124 determines which mirror 303 exists in the effectivevision range based on, for example, the mirror position corresponding tothe model of the driver's vehicle 300. If any mirror 303 is located inthe effective vision range, control is passed to step S9. No mirror 303is located in the effective vision range, control is passed to step S12.

Step S9: The mirror picture generation unit 122 reads the mirror visionarea of the corresponding mirror from the correspondence table DB 131based on the line-of-sight origin P and the mirror 303 located in theeffective vision range. The information about the mirror 303 located inthe effective vision range is included in the line-of-sight data in theline-of-sight data DB 130. The mirror picture generation unit 122extracts each picture corresponding to each mirror vision area.Furthermore, the mirror picture generation unit 122 processes theextracted picture into a mirror picture which may be displayed in themirror display area 266 of the display 255, and stores the picture inthe mirror picture DB 128.

Step S10: The cutoff information calculation unit 123 reads the mirrorcutoff information about the corresponding mirror from thecorrespondence table DB 131 based on the line-of-sight origin P and themirror 303 in the effective vision range, and stores the information inthe cutoff information DB 129.

Step S11: The display picture generation unit 125 reads the mirrorpicture in the mirror picture DB 128 and the mirror cutoff informationin the cutoff information DB 129 for the target frame number i, combinesthe mirror picture with the mirror cutoff information, and generates amirror display picture.

Step S12: The display picture generation unit 125 reads a mirror displayarea from the correspondence table DB 131 based on the line-of-sightorigin P and the line-of-sight vector. Next, the display picturegeneration unit 125 generates a display picture by superposing a mirrordisplay picture on the car window display picture based on the mirrordisplay area.

Step S13: If the frame of the frame number i is the final frame, theprocess terminates. Otherwise, control is returned to step S2.

(5) Effect of Operation

The driving picture processing device 100 may reflect the area in whichthe line of sight of the driver is cut off by a car body such as apillar and so on in the process above. That is, a display picture whichis actually to be visually confirmed by the driver may be generated.Therefore, the viewer of the display picture may grasp the actual statein which a certain area is a hidden by a car body such as a pillar, orin which a dangerous driving has taken place by a dead area by viewingthe display picture whose cutoff information is reflected using thedrive training terminal 250. Thus, the safe drive training may beeffectively performed.

Furthermore, since a display picture is a picture having the line ofsight of a driver at the center, the viewer of the display picture mayview an object by feeling as if the viewer were practically driving avehicle. Especially, when the viewer views the display picture ofperforming dangerous driving, the viewer may grasp the state in whichthe driver was driving the vehicle during the dangerous driving, and mayobtain the feeling of actually performing the dangerous driving.Therefore, effective safe drive training may be performed by providing astrong impression for a viewer about the dangerous driving in a specificsituation, thereby effectively utilizing the training in practicaldriving.

Furthermore, when there is a mirror in the effective vision range, notonly the car window display picture but also a mirror display picturefor observation by the driver through the mirror may be included in thedisplay picture. Thus, the viewer of the display picture may confirm notonly the periphery situation observed by the driver through the carwindow, but also the periphery situation which may be observed throughthe mirror in the effective vision range. Thus, safe drive training maybe realized by evaluating the line of sight of the driver and thedriving state based on all situations in which the driver actuallyperforms the observation.

(6) Variation Example (6-1) Variation Example 1

In the first embodiment described above, as illustrated in FIGS. 25through 27, the display picture is displayed on the display 255 so thatthe center of the end of the line of sight of the driver may bepositioned at the center of the display 255. Thus, although theline-of-sight origin P and the line-of-sight vector of the driverchange, the center of the line of sight is fixed to the central portionof the display 255. On the other hand, the picture in the effectivevision range of the driver moves depending on the center of the line ofsight as illustrated in FIGS. 25 and 26.

However, in the present variation example, as illustrated in FIG. 33described below, the vision area of the display picture displayed on thedisplay 255 is fixed on the 3-dimensional projection surface 400. Thevision area is referred to as a fixed vision area in the presentvariation example. In the present variation example, a line-of-sightlocus 281 of a driver in each frame is displayed on the display 255. Aline-of-sight locus generation unit in the scope of the claims of thepatent is included in the line-of-sight processing unit 124.

(a) Fixed Vision Area

A fixed vision area 400 fix is first described below with reference toFIG. 31. FIG. 31 is an explanatory view of the positions of the fixedvision area on the 3-dimensional projection surface 400, and the carwindow picture and the mirror picture. For example, assume that, among aspecified number of frames, the line-of-sight vector has moved asillustrated by the line-of-sight vector 150 a 1, 150 a 2, and 150 a 3using the line-of-sight origin P as an origin. The front window picture400F is associated as a picture in the forward direction indicated bythe line-of-sight vector 150 a 1 from the line-of-sight origin P.Similarly, the right car window picture 400R and the left car windowpicture 400L are associated as pictures in the direction indicated bythe line-of-sight vector 150 a 2 and 150 a 3 from the line-of-sightorigin P. The fixed vision area 400 fix is set so that the driverincludes the picture which may be visually confirmed by the driver amongthe specified number of frames. That is, the fixed vision area 400 fixis set so that the right car window picture 400R and the left car windowpicture 400L may be included.

Furthermore, as illustrated in FIG. 31, the back mirror picture 400MB isassociated with the back mirror 303B. The right mirror picture 400MR isassociated with the right door mirror 303R. The left mirror picture400ML is associated with the left door mirror 303L.

The fixed vision area 400 fix may be a constantly fixed area, or dependson the line-of-sight origin P and the line-of-sight vector. For example,the size and the position of the fixed vision area 400 fix may depend onthe average sight-of-line origin Pav and the average sight-of-linevector among the specified number of frames. For example, theline-of-sight processing unit 124 calculates the average sight-of-lineorigin Pav by averaging the line-of-sight origin P among the specifiednumber of frames, and calculates the average sight-of-line vector byaveraging the line-of-sight vector among the specified number of frames.Each correspondence table DB 131 stores the correspondence among theaverage sight-of-line origin Pav and the average sight-of-line vector,the fixed vision area 400 fix on the 3-dimensional projection surface400, and the intersection SP of the average sight-of-line vector fromthe average sight-of-line origin Pav and the 3-dimensional projectionsurface 400. Therefore, the car window picture generation unit 121 maydetermine the fixed vision area 400 fix from each car window picturegeneration unit 121 based on the average sight-of-line origin Pav andthe average sight-of-line vector.

(b) Relationship Between the Car Window Picture and the Mirror Picture,and the Display Area of Display

FIG. 32 is an explanatory view of the relationship between the carwindow picture and the mirror picture on the 3-dimensional projectionsurface, and the display area of the display. The display area of thedisplay 255 includes the car window display area 265 and the mirrordisplay area 266. In the present embodiment, the relationship inposition between the car window display area 265 and the mirror displayarea 266. In the present variation example, the positional relationshipbetween the car window display area 265 and the mirror display area 266is fixed, and is set as a specified positional relationship.

The car window display area 265 displays the car window display pictureconfigured by a car window picture corresponding to the fixed visionarea 400 fix and the car window cutoff information. The mirror displayarea 266 includes the back mirror display area 266B, the right mirrordisplay area 266R, and the left mirror display area 266L. The backmirror display area 266B, the right mirror display area 266R, and theleft mirror display area 266L display the respective mirror displaypictures configured by the respective mirror pictures 400MB, 400MR, andthe 400MB, and the mirror cutoff information of each mirror.

(c) Flow of Processes

Described briefly below is the flow of the following processes.

The car window picture generation unit 121 projects the peripheralinformation about a target frame on the 3-dimensional projection surface400. Next, the car window picture generation unit 121 extracts a picturecorresponding to the fixed vision area 400 fix from the 3-dimensionalprojection surface 400 on which the peripheral picture is projected,processes the picture as a car window picture which may be displayed onthe display 255, and stores the resultant picture in the window pictureDB 127.

The correspondence table DB 131 stores the car model, the line-of-sightorigin P, and the line-of-sight vector 150 a as associated with the carwindow cutoff information fix about the car body of the driver's vehicle300 which cuts off the car window line-of-sight of the driver. The carwindow cutoff information fix indicates the cutoff of the sight-of-lineof the driver.

The cutoff information calculation unit 123 reads the car window cutoffinformation fix from the correspondence table DB 131 based on the modelof the driver's vehicle, the line-of-sight origin P, and theline-of-sight vector 150 a, and stores the information in the cutoffinformation DB 129.

The display picture generation unit 125 generates a car window displaypicture corresponding to the fixed vision area 400 fix for the targetframe based on the car window picture in the fixed vision area 400 fixand the car window cutoff information fix.

In addition, the line-of-sight processing unit 124 calculates theeffective vision range based on the line-of-sight origin P and theline-of-sight vector 150 a, and the effective vision angle θe of aspecified angle, and determines which mirror 303 exists in the effectivevision range. The line-of-sight processing unit 124 also refers to FIG.13 described above, and calculates the intersection SPa between theline-of-sight vector 150 a extending from the line-of-sight origin P,and the 3-dimensional projection surface 400 based on the line-of-sightorigin P and the line-of-sight vector 150 a. Furthermore, theline-of-sight processing unit 124 coordinate-converts the intersectionSPa on the 3-dimensional projection surface 400 into the point on thedisplay 255, thereby calculating the sight-of-line locus.

As in the first embodiment above, the mirror picture generation unit 122reads the mirror vision area of the mirror in the effective vision rangefrom the correspondence table DB 131 in FIGS. 16 A and 16B based on theline-of-sight origin P and the mirror 303 located in the effectivevision range. The mirror picture generation unit 122 extracts eachpicture corresponding to each mirror vision area from the 3-dimensionalprojection surface 400 on which the peripheral picture is projected,processes the extracted picture into a mirror picture, and stores theresultant picture in the mirror picture DB 128.

The cutoff information calculation unit 123 reads the mirror cutoffinformation about the corresponding mirror from the correspondence tableDB 131 based on the line-of-sight origin P, and the mirror 303 existingin the effective vision range as with the first embodiment, and storesthe information in the cutoff information DB 129.

The display picture generation unit 125 generates a mirror displaypicture according to the mirror picture and the mirror cutoffinformation for a target frame. Furthermore, the display picturegeneration unit 125 superposes the mirror display picture on the carwindow display picture based on specified position relationship, andfurther superposes a sight-of-line locus, thereby generating a displaypicture.

(d) Example of Display Picture

In the processes above, for example, the display picture as illustratedin, for example, FIG. 33 is displayed on the display 255. FIG. 33 is anexample of a display picture.

In FIG. 33, the car window display picture corresponding to the fixedvision area 400 fix is displayed on the car window display area 265. Thecar window display picture includes the car window cutoff informationconfigured by the car body area 280F by the front pillar 307F, the carbody area 280R by the right pillar 307R, the car body area 280L by theleft pillar 307L, and the car body area 280D by the dashboard 301.

In the example illustrated in FIG. 33, only the back mirror 303B existsin the effective vision range of the driver, and the back mirror picture400MB is displayed in the back mirror display area 266B. Since otherright door mirror 303R and left door mirror 303L are not located in theeffective vision range, the right mirror display area right mirrordisplay area 266R and the left mirror display area 266L display nopictures.

Furthermore, in FIG. 33, the line-of-sight locus 281 of the driver isdisplayed. Thus, since the sight-of-line locus of the driver issuperposed on the display picture, the viewer may grasp what objectother than the driver's vehicle the driver has or has not visuallyconfirmed during the driving of the driver's vehicle. Thus, for example,the cause of the dangerous driving, for example, due to no recognizingan object to be visually confirmed, and so on, is analyzed, therebyutilizing the obtained data in safe drive training.

With the generated display picture, the range of the fixed vision area400 fix on the 3-dimensional projection surface 400 does not change.However, the line-of-sight locus 281 is generated depending on themovement of the sight-of-line of the driver, and the car window cutoffinformation changes. Since the car window cutoff information changes,the car body area of, for example, a pillar and so on also changeddepending on the movement of the sight-of-line as illustrated in, forexample, FIG. 33. Also depending on the movement of the sight-of-line ofthe driver, the mirror display area 266 in which the mirror displaypicture is displayed changes.

(6-2) Variation Example 2

Depending on the level of the tension during the driving and theconcentration on the driving, the vision range in which the driver mayvisually confirm changes. For example, when the driver is nervous orconcentrates his or her attention too much on one object, the visionrange of the driver tends to be narrowed. Then, according to the presentvariation example, the vision range is calculated according to thebiological information such as the diameter of the pupils, the number ofpulses, the state of the pulses, the amount of perspiration, theretention time of the sight-of-line, and so on, and a display picture isprocessed depending on the vision range.

FIG. 34 is an example of a block diagram of the functional configurationof each device relating to the variation example 2. The functionalconfiguration of the present variation example includes a visioncalculation unit 132 in addition to the functional configuration of thefirst embodiment in FIG. 8.

The vision range may be calculated according to the biologicalinformation such as the diameter of the pupils, the number of pulses,the state of the pulses, the amount of perspiration, the retention timeof the sight-of-line, and so on. The biological information may bedetected by each detection unit.

The diameter of pupils may be measured by the line-of-sight detectionunit 221. For example, the line-of-sight detection unit 221 acquires thepicture of an eye, extracts the image of the pupil, and measures thediameter of the pupil. Otherwise, the line-of-sight detection unit 221emits light such as infrared and so on, and measures the diameter of thepupil based on the wave reflected by the eye.

The number of pulses may be measured by a measure attached to the handle302 based on the blood flow through the hands at the handle 302. Themeasure has a plus electrode or a minus electrode at the positions ofthe right and left hands on the steering wheel.

The amount of perspiration may be measured by the measure attached tothe handle 302 based on the perspiration emitted from the hands on thehandle 302.

The retention time of the sight-of-line may be obtained by calculatingthe time in which the sight-of-line is held in each direction of vectorbased on the line-of-sight origin P and the line-of-sight vector 150 a.

The information for calculation of the vision range is not limited tothe information described above, but may be various types of biologicalinformation such as blood pressure and so on.

The information for calculation of the vision range is provided for thevision calculation unit 132.

(b) Calculation of Vision Range

The vision calculation unit 132 calculates the vision range according tothe information for calculation of the above-mentioned vision range. Forexample, the correspondence table DB 131 stores the correspondencebetween the diameter of the pupils, the number of pulses, the state ofthe pulses, the amount of perspiration, the retention time of thesight-of-line, and so on, as associated with the vision range. Forexample, the smaller the diameter of the pupils, the smaller the visionrange. Furthermore, the larger the number of pulses, the smaller thevision range. The vision calculation unit 132 refers to thecorrespondence, and calculates the vision range. The vision range isexpressed by the coordinates in the display area of the display 255.

(c) Process of the Picture Depending on the Vision Range

The display picture generation unit 125 acquires the vision range fromthe vision calculation unit 132, and processes the display picture basedon the vision range.

FIG. 35 is an explanatory view of an example of processing a displaypicture. The point SP as the center of the end portion of thesight-of-line of the driver is located at the center of the display areaof the display 255, and the vision range VR including the point SP iscalculated. The vision range VF is, for example, L1 long and L2 wide.The vision range VF is not limited to a rectangle, but may be circular,oval, and so on.

It is assumed that the driver is able to visually confirm the state ofthe periphery of the driver's vehicle in the vision range. On the otherhand, it is assumed that the state of the periphery of the driver'svehicle is not visually confirmed outside the vision range. The displaypicture generation unit 125 performs the process so that the displaypicture may be clearly confirmed in the vision range VF, and the displaypicture may be faded in the display area outside the vision range VF.

With the above-mentioned processing on the display picture, the state ofthe observation by the driver may be estimated and reproduced. Thus, theviewer may confirm the display picture depending on the vision range ofthe driver. Thus, for example, when a target which has caused dangerousdriving by, for example, a narrow vision, due to not grasping the targetby the driver may be effectively analyzed using the display pictureabove.

(6-3) Variation Example 3

In the first embodiment above, the driving picture processing device 100projects a peripheral picture on the 3-dimensional projection surface400, extracts a car window picture and a mirror picture from theperipheral picture on the 3-dimensional projection surface 400, andprocesses the pictures so that they may be displayed on the display 255.However, the driving picture processing device 100 may generate a carwindow picture and a mirror picture which may be displayed on thedisplay 255 from the peripheral picture acquired from each of thecameras 205 a through 205 d. Therefore, for example, the correspondencetable DB 131 stores for each line-of-sight origin P and line-of-sightvector the correspondence between the coordinates of each pixelconfiguring the picture corresponding to the car window line-of-sight inthe peripheral pictures with the coordinates on the display area of thedisplay 255. The car window picture generation unit 121coordinate-converts the picture data corresponding to the sight-of-lineof the driver in the peripheral pictures into the display area of thedisplay 255 from the peripheral information acquisition equipment 205based on the specified line-of-sight origin P and line-of-sight vector,and the correspondence. Thus, a car window picture corresponding to theline-of-sight origin P and the line-of-sight vector of the driver may begenerated.

The same holds true with the mirror picture. For example, eachcorrespondence table DB 131 stores the correspondence between thecoordinates of each pixel configuring the picture corresponding to themirror line-of-sight in the peripheral pictures as associated with thecoordinates in the display area of the display 255 in association withthe mirror information and the line-of-sight origin P. The mirrorpicture generation unit 122 generates a mirror picture based on thecorrespondence for the mirror 303 in the effective vision range.

(6-4) Variation Example 4

The driving picture processing device 100 according to the firstembodiment superposes the car window display picture and the mirrordisplay picture to generate a display picture as illustrated in FIGS. 28and 29. However, the driving picture processing device 100 may generateonly the car window display picture as a display picture, or generateonly the mirror display picture as a display picture.

Second Embodiment

The driving picture processing device 100 according to the firstembodiment acquires the peripheral information and the line-of-sightdata around the driver's vehicle from an external informationacquisition device 200. On the other hand, the driving pictureprocessing device 100 according to the second embodiment acquires theinformation. Described below are the differences from the firstembodiment.

The configuration of the driving picture processing device 100 accordingto the second embodiment is described below. FIG. 36 is an example of ablock diagram of the hardware configuration of the driving pictureprocessing device.

The driving picture processing device 100 has, for example, the CPU 101,the ROM 102, the RAM 103, the input/output equipment I/F 104, and thecommunication I/F 108. They are interconnected through the bus 109.

The input/output equipment I/F 104 is connected to the input/outputequipment such as the display 105, the mouse 106, the keyboard 107, theperipheral information acquisition equipment 205, the line-of-sightdetection equipment 206, and so on.

The functional configuration of the driving picture processing device100 is described below. FIG. 37 is an example of a block diagram of thefunctional configuration of the driving picture processing deviceaccording to the second embodiment. The driving picture processingdevice 100 according to the second embodiment includes the peripheralinformation acquisition unit 220 and the line-of-sight detection unit221 in addition to the driving picture processing device 100 accordingto the first embodiment. Since the driving picture processing device 100according to the second embodiment does not transmit or receive data,commands, and so on to or from the information acquisition device 200,the transmission/reception units 120 and 222 and the acquired data DB223 are omitted here. The processing of each function is similar to thataccording to the first embodiment.

Other configurations are similar to those according to the firstembodiment. Furthermore, in the second embodiment, a variation exampleof the first embodiment may be applied.

Other Embodiments

A computer program for directing a computer to perform the method aboveand a computer-readable recording medium which stores the computerprogram are included in the scope of the present invention. Thecomputer-readable recording medium may be, for example, a flexible disk,a hard disk, CD-ROM (compact disc read only memory), an MO (magnetooptical disk), a DVD, DVD-ROM, DVD-RAM (DVD: random access memory), BD(blue-ray disk), USB memory, semiconductor memory, and so on. Thecomputer program is not limited to that stored on the recording medium,but may be transmitted through an electric communication circuit, awireless or cable communication circuit, a network represented by theInternet. However, a computer-readable recording medium does not includea carrier wave in which a computer program is embedded. A computerprogram transmitted by being embedded in a carrier wave, which is acomputer readable recording medium which stores the program is arecording medium having a physical entity to be reproduced in arecording medium reading device which is connected to a transmittingcomputer.

The present invention may provide a picture processing device, a pictureprocessing method, and a picture processing program which generate apicture for generating a picture obtained by a driver.

All examples and conditional language recited herein are intended forpedagogical purposes to aid the reader in understanding the inventionand the concepts contributed by the inventor to furthering the art, andare to be construed as being without limitation to such specificallyrecited examples and conditions, nor does the organization of suchexamples in the specification relate to a showing of the superiority andinferiority of the invention. Although the embodiments of the presentinvention have been described in detail, it should be understood thatthe various changes, substitutions, and alterations could be made heretowithout departing from the spirit and scope of the invention.

What is claimed is:
 1. A picture processing device, comprising: apicture acquisition unit configured to acquire peripheral informationincluding a picture obtained by shooting a periphery of a driver'svehicle; a line-of-sight acquisition unit configured to acquire aline-of-sight origin and a direction of the line of sight of a driver ofthe driver's vehicle; a line-of-sight picture generation unit configuredto generate from the peripheral information a line-of-sight picturecorresponding to the line-of-sight origin; a cutoff informationcalculation unit configured to calculate cutoff information includingthe car body area or a car body picture of the driver's vehicle whichcuts off a line of sight of the driver based on the line-of-sightorigin; and a display picture generation unit configured to generate adisplay picture according to the line-of-sight picture and the cutoffinformation.
 2. The device according to claim 1, wherein: the directionof the line of sight includes a car window line-of-sight direction fromthe line-of-sight origin through a window of the driver's vehicle; theline-of-sight picture includes a car window picture corresponding to theline-of-sight origin and the car window line-of-sight direction; thecutoff information includes car window cutoff information containing acar body area and/or a car body picture of the driver's vehicle aboutcutoff of the sight-of-line in the car window line-of-sight direction ofthe driver; the display picture includes a car window display picturefor observation by the driver through a window of the driver's vehicle;the line-of-sight picture generation unit comprises a window picturegeneration unit which generates the car window picture from a peripheralpicture of the driver's vehicle; the cutoff information calculation unitcalculates the car window cutoff information based on the line-of-sightorigin; and the display picture generation unit generates the car windowdisplay picture according to the car window picture and the car windowcutoff information.
 3. The device according to claim 1, wherein: theline-of-sight picture includes a mirror picture corresponding to avisual mirror confirmation range in which the driver may perform avisual confirmation through at least one mirror of the driver's vehicle;the cutoff information includes mirror cutoff information containing acar body area and/or a car body picture of the driver's vehicle whichcuts off a mirror line-of-sight of the driver to the visual mirrorconfirmation range; the display picture includes a mirror displaypicture for observation by the driver through at least one mirror of thedriver's vehicle; the device further comprises a sight-of-lineprocessing unit configured to calculate a effective vision range havinga specified angle range in which the driver may perform a visualconfirmation based on the line-of-sight origin and the direction of theline of sight, and determine whether or not there is at least one mirrorof the driver's vehicle in the effective vision range; the line-of-sightpicture generation unit comprises a mirror picture generation unitconfigured to calculate the visual mirror confirmation range in whichthe driver may perform a visual confirmation through the mirroraccording to mirror information including the line-of-sight origin and aposition of the mirror when at least one mirror is located in theeffective vision range, and generate the mirror picture corresponding tothe visual mirror confirmation range from a peripheral picture of thedriver's vehicle; the cutoff information calculation unit calculates themirror cutoff information based on the line-of-sight origin and thevisual mirror confirmation range; and the display picture generationunit generates the mirror display picture according to the mirrorpicture and the mirror cutoff information.
 4. The device according toclaim 3, wherein the display picture generation unit superposes themirror display picture corresponding to a mirror in the effective visionrange on the car window display picture when at least one mirror islocated in the effective vision range.
 5. The device according to claim1, further comprising a vision calculation unit configured to calculatea vision range of the driver according to at least one piece ofbiological information including a pupil, a number of pulses,perspiration, and a retention time in the direction of the line ofsight; and the display picture generation unit processes the displaypicture based on the vision range.
 6. The device according to claim 5,wherein the display picture generation unit fades an area outside thevision range in the display picture.
 7. The device according to claim 1,further comprising a line-of-sight locus generation unit configured togenerate a sight-of-line locus in the line-of-sight picture which isdesignated by the line-of-sight origin and the direction of the line ofsight; and the display picture generation unit superposes thesight-of-line locus on the display picture.
 8. The device according toclaim 1, wherein the line-of-sight origin is an average line-of-sightorigin obtained by averaging line-of-sight origin of the driver in aspecified time; and the direction of the line of sight is an averagedirection of the line of sight obtained by averaging a direction of theline of sight in the specified time.
 9. The device according to claim 1,wherein the line-of-sight picture generation unit generates a projectionpicture by projecting a peripheral picture of the driver's vehicle on a3-dimensional projection surface as a virtual space, and generates asthe sight-of-line picture a picture corresponding to the line-of-sightorigin and the direction of the line of sight in the projection picture.10. The device according to claim 1, wherein the display picturegeneration unit generates the display picture by removing a picture ofan area of a body of the driver's vehicle included in the cutoffinformation from the line-of-sight picture.
 11. The device according toclaim 1, wherein the display picture generation unit generates thedisplay picture by superposing a car body picture of the driver'svehicle included in the cutoff information on the line-of-sight picture.12. A non-temporary medium which stores a picture processing programused to direct a processor to perform a process, the process comprising:acquiring peripheral information including a picture obtained byshooting a periphery of a driver's vehicle; acquiring a line-of-sightorigin and a direction of the line of sight of a driver of the driver'svehicle; generating from the peripheral information a line-of-sightpicture corresponding to the line-of-sight origin; calculating cutoffinformation including the car body area or a car body picture of thedriver's vehicle which cuts off a line of sight of the driver based onthe line-of-sight origin; and generating a display picture according tothe line-of-sight picture and the cutoff information.
 13. A method forallowing a processor to perform a process, the method comprising:acquiring peripheral information including a picture obtained byshooting a periphery of a driver's vehicle; acquiring a line-of-sightorigin and a direction of the line of sight of a driver of the driver'svehicle; generating from the peripheral information a line-of-sightpicture corresponding to the line-of-sight origin; calculating cutoffinformation including the car body area or a car body picture of thedriver's vehicle which cuts off a line of sight of the driver based onthe line-of-sight origin; and generating a display picture according tothe line-of-sight picture and the cutoff information.